ES|QL - Full text functions accept null as field parameter #137430

carlosdelest · 2025-10-31T10:03:08Z

Full text functions that take a field as a parameter should allow null as a field parameter. This is necessary as the field may not be present on an index mapping, and the local optimizer replaces it by null.

We allow null as a field in FullTextFunctions, meaning that they are nullable and will be replaced by NULL.

This PR also refactors full text functions that depend on a single field (MATCH, MATCH_PHRASE, KNN) to use a common superclass (SingleFieldFullTextFunction) that contains the common field logic.

… and add the FoldNull rule to LocalLogicalPlanOptimizer

…down to an index that doesn't contain the field

elasticsearchmachine · 2025-10-31T10:03:34Z

Hi @carlosdelest, I've created a changelog YAML for you.

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

…-functions-accept-null # Conflicts: # x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizerTests.java

carlosdelest · 2025-10-31T10:04:53Z

...ugin/esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/Match.java

@@ -300,39 +286,7 @@ public final void writeTo(StreamOutput out) throws IOException {

    @Override
    protected TypeResolution resolveParams() {
-        return resolveField().and(resolveQuery())


Refactoring - all of these are moved up to SingleFieldFullTextFunction

carlosdelest · 2025-10-31T10:05:57Z

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

+    }
+
+    @Override
+    public boolean foldable() {


This is what changed on this PR besides all the refactoring. When the FTF field is null, the FTF is foldable to NULL, and nullable.

carlosdelest · 2025-10-31T11:32:32Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java

                new InferNonNullAggConstraint(),
-                new ReplaceDateTruncBucketWithRoundTo()
+                new ReplaceDateTruncBucketWithRoundTo(),
+                new FoldNull()


@alex-spies I added foldNull to local logical optimization, so full text functions can be folded to null when null has been replaced. Does it look OK to you?

I think this should be beneficial, yes. Not sure why we didn't have it earlier.
Do you think you could please add one/some LocalLogicalPlanOptimizerTests with maybe some "easy" other function(s) (string or math?) to check that they locally now get folded too. This should add a tiny bit of performance to other functions too.

the FoldNull rule is in the localOperators batch, that is applied after the localRewrites batch. Is it too late to wait for it to be executed in localOperators? I wonder why we need a duplicate of it here?

Oh, I see - I didn't understand that this rule was already used as part of localOperators(). Removing it, thanks @fang-xing-esql !

carlosdelest · 2025-10-31T14:34:59Z

...lasticsearch/xpack/esql/optimizer/rules/logical/PushDownFilterAndLimitIntoUnionAllTests.java

+            """);
+
+        // Limit[1000[INTEGER],false,false]
+        Limit limit = as(plan, Limit.class);


@fang-xing-esql , now FTFs can be pushed down - when a field does not exist in the index, it will become NULL instead.

Does this make sense from the union all perspective?

I found a potential issue here and I'll provide a reproducer.

The query in the test failed with VerificationException because the full text function does not apply to one of the subquery(branch), the field referenced in the full text function is not an index field, it doesn't throw VerificationException in this PR, however I run into this error when the query is executed. Shall we still throw VerificationException for this kind of situations? Or prune the entire branch when the field does not exist in that branch, a related comment is here.

+ curl -u elastic:password -X POST 'localhost:9200/_query?format=txt&pretty' -H 'Content-Type: application/json' '-d { "query": "from sample_data_1, (from books) metadata _score | WHERE author:\"Bradbury\" | sort _score desc, author" } ' { "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "Unrecognized class type class org.elasticsearch.xpack.esql.core.expression.Literal" } ], "type" : "illegal_argument_exception", "reason" : "Unrecognized class type class org.elasticsearch.xpack.esql.core.expression.Literal" }, "status" : 400 }

The mapping and data a like below.

curl -u elastic:password -X POST "localhost:9200/books/_doc?pretty" -H 'Content-Type: application/json' -d' {"name": "Snow Crash", "author": "Neal Stephenson", "release_date": "1992-06-01", "page_count": 470, "*name": "*Snow Crash"} ' curl -u elastic:password -X POST "localhost:9200/_bulk?refresh&pretty" -H 'Content-Type: application/json' -d' { "index" : { "_index" : "books" } } {"name": "Revelation Space", "author": "Alastair Reynolds", "release_date": "2000-03-15", "page_count": 585, "*name": "*Revelation Space"} { "index" : { "_index" : "books" } } {"name": "1984", "author": "George Orwell", "release_date": "1985-06-01", "page_count": 328, "*name": "*1984"} { "index" : { "_index" : "books" } } {"name": "Fahrenheit 451", "author": "Ray Bradbury", "release_date": "1953-10-15", "page_count": 227, "*name": "*Fahrenheit 451"} { "index" : { "_index" : "books" } } {"name": "Brave New World", "author": "Aldous Huxley", "release_date": "1932-06-01", "page_count": 268, "*name": "*Brave New World"} { "index" : { "_index" : "books" } } {"name": "The Handmaids Tale", "author": "Margaret Atwood", "release_date": "1985-06-01", "page_count": 311, "*name": "*The Handmaids Tale"} ' curl -u elastic:password -X PUT "localhost:9200/sample_data_1?pretty" -H 'Content-Type: application/json' -d' { "settings" : {"mode" : "lookup"}, "mappings": { "properties": { "client.ip": {"type": "ip"}, "message": {"type": "text"}, "test_date": {"type": "date"}, "event.duration": {"type": "long"}, "mv": {"type": "long"} } } } ' curl -X PUT "localhost:9200/sample_data_1/_bulk?refresh&pretty" -H 'Content-Type: application/json' -d' {"index": {}} {"@timestamp": "2023-10-23T12:15:03.360Z", "client.ip": "172.21.2.162", "message": "Connected to 10.1.0.3", "event.duration": 3450233, "test_date": "1985-11-21T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T12:27:28.948Z", "client.ip": "172.21.2.113", "message": "Connected to 10.1.0.2", "event.duration": 2764889, "test_date": "1985-11-22T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T13:33:34.937Z", "client.ip": "172.21.0.5", "message": "Disconnected", "event.duration": 1232382, "test_date": "1985-11-23T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T13:51:54.732Z", "client.ip": "172.21.3.15", "message": "Connection error", "event.duration": 725448, "test_date": "1985-11-24T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T13:52:55.015Z", "client.ip": "172.21.3.15", "message": "Connection error", "event.duration": 8268153, "test_date": "1985-11-25T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T13:53:55.832Z", "client.ip": "172.21.3.15", "message": "Connection error", "event.duration": 5033755, "test_date": "1985-11-26T00:00:00Z", "mv": [1.0, 2.0]} {"index": {}} {"@timestamp": "2023-10-23T13:55:01.543Z", "client.ip": "172.21.3.15", "message": "Connected to 10.1.0.1", "event.duration": 1756467, "test_date": "1985-11-27T00:00:00Z", "mv": [1.0, 2.0]} '

elasticsearchmachine · 2025-10-31T14:35:33Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

elasticsearchmachine · 2025-10-31T14:35:34Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2025-10-31T14:35:34Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

kderusso

Changes look good to me, but will defer to someone with more ES|QL experience to 👍 😁

kderusso · 2025-10-31T17:54:52Z

...lasticsearch/xpack/esql/optimizer/rules/logical/PushDownFilterAndLimitIntoUnionAllTests.java

    public void testFullTextFunctionCannotBePushedDownPastUnionAll() {
        assumeTrue("Requires subquery in FROM command support", EsqlCapabilities.Cap.SUBQUERY_IN_FROM_COMMAND.isEnabled());
-        VerificationException e = expectThrows(VerificationException.class, () -> planSubquery("""
+        var plan = planSubquery("""


Do we need to specify a new capability for this test?

We don't, as the MATCH function is already released (so it's not conditionally added for release / snapshots) and this test is not executed for BwC purposes.

kderusso · 2025-10-31T18:09:59Z

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

+ * - Field verification
+ * - Serialization patterns
+ */
+public abstract class SingleFieldFullTextFunction extends FullTextFunction


Nice refactoring!

carlosdelest · 2025-11-04T11:36:52Z

@elasticsearchmachine run elasticsearch-ci/bwc-snapshots-part3

bpintea

Left a few comments, but after those it should look good.
I think some simple CSV-spec tests would be good to have too, just to check that they work with literals, centrally too. (MATCH(null, ..), ..| EVAL x = null | WHERE MATCH(x, ...)).

Would be great to wait for @fang-xing-esql check too.

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

bpintea · 2025-11-04T11:26:18Z

...rc/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/FullTextFunction.java

+    protected FieldAttribute fieldAsFieldAttribute(Expression field) {
        Expression fieldExpression = field;
        // Field may be converted to other data type (field_name :: data_type), so we need to check the original field
        if (fieldExpression instanceof AbstractConvertFunction convertFunction) {


A somewhat lateral note: we could have an unpacking loop here, for the (more likely automated query generation) case of nested conversions, like: field::INTEGER::LONG.

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

...esql/src/main/java/org/elasticsearch/xpack/esql/expression/function/fulltext/MultiMatch.java

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java

bpintea · 2025-11-04T14:26:04Z

...lasticsearch/xpack/esql/expression/function/fulltext/AbstractMatchFullTextFunctionTests.java

        AbstractMatchFullTextFunctionTests.addQueryAsStringTestCases(suppliers);
        AbstractMatchFullTextFunctionTests.addStringTestCases(suppliers);
-        return suppliers;
+        return addNullFieldTestCases(suppliers);


I think these aren't added to all functions, like MatchPhraseTests (possibly others, Knn?), right?
Raising this since MatchPhrase.FIELD_DATA_TYPES doesn't include NULL.

Good call! Done in 845cd35

bpintea · 2025-11-04T14:30:42Z

...gin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalLogicalPlanOptimizer.java

                new InferNonNullAggConstraint(),
-                new ReplaceDateTruncBucketWithRoundTo()
+                new ReplaceDateTruncBucketWithRoundTo(),
+                new FoldNull()


I think this should be beneficial, yes. Not sure why we didn't have it earlier.
Do you think you could please add one/some LocalLogicalPlanOptimizerTests with maybe some "easy" other function(s) (string or math?) to check that they locally now get folded too. This should add a tiny bit of performance to other functions too.

Co-authored-by: Bogdan Pintea <[email protected]>

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

fang-xing-esql

Thank you @carlosdelest, and for taking into account subqueries as well! Agreed with @bogdan, it will be great to add some additional tests(verify the logical plan and some CsvTests to validate query results) that have these full text functions with null as input field. If the full text functions are used in filters, the PruneFilters rule might be able to replace its children plans with a LocalRelation, that is a sign that the filter on full text function + null fields is applied successfully.

fang-xing-esql · 2025-11-04T19:55:35Z

...a/org/elasticsearch/xpack/esql/expression/function/fulltext/SingleFieldFullTextFunction.java

+    @Override
+    public Object fold(FoldContext ctx) {
+        // We only fold when the field is null (it's not present in the mapping), so we return null
+        return Literal.NULL;


I wonder if we should just return a null here, instead of wrapping a Literal around it? A lot of implementations of fold return a plain null.

elasticsearchmachine added 10 commits October 29, 2025 18:43

Remove null check for fields in FTFs

1364419

First version - allow nulls as params, implement nullable() in Match,…

bd00de1

… and add the FoldNull rule to LocalLogicalPlanOptimizer

Introduce base class for Match and MatchPhrase

20b202d

Make Knn a subclass of SingleFieldFullTextFunction, refactor

c7bda0f

Refactor

7f5f5c6

Fix PushDownFilterAndLimitIntoUnionAllTests, now match can be pushed …

8fa9556

…down to an index that doesn't contain the field

Spotless

bd6e99d

Refactor

8ce4820

Fix error tests

1d98c74

Add check for single unique signature

ec81181

carlosdelest added >bug :Analytics/ES|QL AKA ESQL :Search Relevance/ES|QL Search functionality in ES|QL v9.3.0 labels Oct 31, 2025

Update docs/changelog/137430.yaml

336b33c

carlosdelest added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) :SearchOrg/Relevance Label for the Search (solution/org) Relevance team Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch labels Oct 31, 2025

elasticsearchmachine added 5 commits October 31, 2025 11:51

Fix tests and add null cases

959cac7

Merge remote-tracking branch 'carlosdelest/bugfix/esql-full-text-func…

21daf3d

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

Add test cases

b737b03

Fix tests

f68b569

carlosdelest commented Oct 31, 2025

View reviewed changes

carlosdelest marked this pull request as ready for review October 31, 2025 14:35

elasticsearchmachine added the Team:Search - Relevance The Search organization Search Relevance team label Oct 31, 2025

carlosdelest requested review from alex-spies, fang-xing-esql and ioanatia October 31, 2025 14:41

kderusso reviewed Oct 31, 2025

View reviewed changes

julian-elastic mentioned this pull request Nov 3, 2025

ES|QL - Full text functions should work with LOOKUP JOIN #129153

Open

bpintea approved these changes Nov 4, 2025

View reviewed changes

elasticsearchmachine and others added 8 commits November 4, 2025 16:58

Remove FoldNull as it is already applied in localOperators()

2436a22

Apply suggestion from @bpintea

012378b

Co-authored-by: Bogdan Pintea <[email protected]>

Apply suggestion from @bpintea

30a8319

Co-authored-by: Bogdan Pintea <[email protected]>

Apply suggestion from @bpintea

569b084

Co-authored-by: Bogdan Pintea <[email protected]>

Merge remote-tracking branch 'carlosdelest/bugfix/esql-full-text-func…

04b7dd9

…tions-accept-null' into bugfix/esql-full-text-functions-accept-null

Feedback from @bpintea

ad811e5

Add null test cases for Knn and MatchPhrase

845cd35

Merge branch 'main' into bugfix/esql-full-text-functions-accept-null

48fa7c1

fang-xing-esql reviewed Nov 4, 2025

View reviewed changes

ES|QL - Full text functions accept null as field parameter #137430

Are you sure you want to change the base?

ES|QL - Full text functions accept null as field parameter #137430

Conversation

carlosdelest commented Oct 31, 2025

Uh oh!

elasticsearchmachine commented Oct 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bpintea Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 31, 2025

Uh oh!

elasticsearchmachine commented Oct 31, 2025

Uh oh!

elasticsearchmachine commented Oct 31, 2025

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carlosdelest commented Nov 4, 2025

Uh oh!

bpintea left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bpintea Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fang-xing-esql left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

bpintea Nov 4, 2025 •

edited

Loading

fang-xing-esql Nov 4, 2025 •

edited

Loading

bpintea left a comment •

edited

Loading

bpintea Nov 4, 2025 •

edited

Loading